1 research outputs found
DuETT: Dual Event Time Transformer for Electronic Health Records
Electronic health records (EHRs) recorded in hospital settings typically
contain a wide range of numeric time series data that is characterized by high
sparsity and irregular observations. Effective modelling for such data must
exploit its time series nature, the semantic relationship between different
types of observations, and information in the sparsity structure of the data.
Self-supervised Transformers have shown outstanding performance in a variety of
structured tasks in NLP and computer vision. But multivariate time series data
contains structured relationships over two dimensions: time and recorded event
type, and straightforward applications of Transformers to time series data do
not leverage this distinct structure. The quadratic scaling of self-attention
layers can also significantly limit the input sequence length without
appropriate input engineering. We introduce the DuETT architecture, an
extension of Transformers designed to attend over both time and event type
dimensions, yielding robust representations from EHR data. DuETT uses an
aggregated input where sparse time series are transformed into a regular
sequence with fixed length; this lowers the computational complexity relative
to previous EHR Transformer models and, more importantly, enables the use of
larger and deeper neural networks. When trained with self-supervised prediction
tasks, that provide rich and informative signals for model pre-training, our
model outperforms state-of-the-art deep learning models on multiple downstream
tasks from the MIMIC-IV and PhysioNet-2012 EHR datasets.Comment: Accepted at MLHC 2023, camera-ready versio